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alpha levels for their individual statistical tests in order to keep the 
overall alpha value at a reasonable level. A three-step procedure is 
presented that can be used to adjust the alpha levels of the individual 
statistical tests. This procedure requires researchers to: (l) identify the 

appropriate conceptual unit for the error rate (pairwise, experimentwise, and 
familywise) ; (2) determine the number and nature of tests contained in that 

error rate unit; and (3) apply a Bonferroni-type adjustment procedure to the 
various statistical tests contained in each error rate unit. This three-step 
adjustment procedure emphasizes that it is the obligation of the researchers 
to make logical decisions, not mechanical ones, when adjusting Type I error 
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Abstract 

Educational researchers often use multiple statistical tests in their research studies and program 
evaluations. When multiple statistical tests are conducted, the chance that Type 1 errors may be 
committed increases. Thus, the researchers are faced with the task of adjusting the alpha levels 
for their individual statistical tests in order to keep the overall alpha value at a reasonable level. In 
this paper we present a three-step procedure that can be used to adjust the alpha levels of the 
individual statistical tests. This procedure requires researchers to; (a) identify the appropriate 
conceptual unit for the error rate (pairwise, experimentwise, and familywise), (b) determine the 
number and nature of the tests contained in that error rate unit, and (c) apply a Bonferroni-type 
adjustment procedure to the various statistical tests contained in each error rate unit. This three- 
step adjustment procedure emphasizes that it is the obligation of the researchers to make logical 
decisions not mechanical ones when adjusting Type 1 error rates for multiple statistical tests. 
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The Responsibility of Educational Researchers to Make Appropriate Decisions About 
The Error Rate Unit on Which Type 1 Error Adjustments Are Based: 

A Thoughtful Process not a Mechanical One 
As noted by Stevens (1996), it is common for multiple statistical tests to be conducted in 
educational studies and program evaluations. Stevens suggests that multiple statistical tests are 
encountered in studies that researchers may not readily recognize as the type of studies that 
contain multiple statistical tests. These types of studies include: (a) ANOVA designs that contain 
main effects and interaction effects, (b) analyses of multiple dependent variables, (c) analyses of 
multiple regression models, and (d) analyses of numerous correlation coefficients. 

The use of multiple statistical tests may lead to a situation where the chance of committing 
at least one Type 1 error, i.e., a true null hypothesis will be rejected, is significantly increased. To 
understand the impact of multiple statistical tests on the probability of committing at least one 
Type 1 error, assume that the researchers identified the error unit as the error rate per comparison, 
that is, each statistical test is considered as its own unit of error. If a statistical test is conducted 
with a selected alpha level of .05, the probability of committing a Type I error is .05 for that test. 
For two statistical tests with the alpha level set at .05 for each test, however, the probability of at 
least one Type I error approaches .10. The probability of committing at least one Type 1 error for 
m orthogonal statistical tests with the alpha value for a individual test set at can be calculated 
as follows: 

< m 



p(at least one Type I error) = 1 - (1 - ) 



( 1 . 1 ) 
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where mK |„,| is the approximate upper bound on the probability of committing at least one Type 1 
error. If the m statistical tests are not orthogonal, the probability of committing at least one Type 
1 error can be summarized as follows: 

p(at least one Type 1 error) ^ 1 - (1 - ^ m (1.2) 

The empirical research indicates the probability of committing at least one Type 1 error is fairly 
close to 1 - (1 - )"' for statistical tests that are not orthogonal (Toothaker, 1991). 

As revealed by Equations 1 . 1 and 1 .2 and as noted by Kirk ( 1 982), Stevens ( 1 996), Hays 
(1988), Winer, Brown, and Michels (1991), and Toothaker (1991), the chance of committing a 
Type 1 error can increase dramatically when multiple statistical tests are conducted. The 
previously mentioned authors provide detailed presentations of numerous methods that can be 
used to adjust the Type 1 error rates for multiple statistical tests. These methods include: 

(a) Fisher’s least significance differences test (1949), (b) Tukey’s HSD test (1953), (c) Spjotvoll 
and Stoline’s modification of the HSD test (1973), (d) Tukey-Kramer modification of the HSD 
test (Tukey, 1953; Kramer, 1956), (e) SchefTe’s test (1953), (f) Browm-Fors>'the BF procedure 
(1974), (g) Newman-Keuls test (Newman, 1939; Keuls, 1952), (h) Duncan’s new multiple range 
test (Duncan, 1955), and (i) Bonferroni’s procedure (Dunn, 1961; Newman & Fry, 1972). 

The selection of an appropriate adjustment technique can be a daunting task for even an 
experienced researcher or program evaluator. More importantly, in today’s advanced computer 
age, it is not uncommon for researchers and program evaluators to mechanically select a Type 1 
error adjustment procedure with little thought being given to how and why the adjustment is being 




made. 
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The purpose of this paper is to present a three-step adjustment approach that can easily be 
applied by researchers and program evaluators to numerous types of multiple statistical testing 
situations. This approach is very generalizable, i.e., it can be applied to studies in which one 
wishes to control for various types of comparisons including painvise, familywise, and 
experimentwise. In addition, it can be used with studies that contain repeated measures, 
covariance analyses, and/or unequal sample sizes. 

We believe that this adjustment procedure has an even more important characteristic than 
those previously mentioned. This three-step procedure requires the researchers to address two 
questions: 

1 . What is the appropriate conceptual error rate unit? 

2. What specific adjustment should be made for each statistical test in that unit in order to 
maintain an acceptable overall alpha level? 

The act of addressing these two questions prevents researchers from mechanically implementing 
Type 1 error adjustments to their multiple statistical tests. Rather, it requires thoughtfulness on 
the part of the researchers. Thus, the use of this procedure should provide researchers and 
program evaluators with a better understanding of their analyses. 

Hypothetical Study 

To illustrate how the three-step procedure can be used by researchers and program 
evaluators, we will refer to a hypothetical study that involves three groups of public school 
teachers. The study is designed to evaluate the impact that three instructional methods have on 
the participants’ teaching efficacy levels. In this study, two elements of teacher efficacy that will 
be recorded for each participant are personal efficacy and teaching efficacy. The personal efficacy 



The Responsibility 6 



scores will measure the degree that a teacher believes she/he can personally impact student 
learning. The teaching efficacy scores will measure the degree that a teacher believes teachers, in 
general, can impact student learning. The participants personal and teaching efficacy levels will 
be measured before and after they were exposed to one of the three instructional methods. The 
post-treatment personal efficacy scores and the post-treatment teaching efficacy scores will serve 
as the dependent variables. 

Various hypotheses related to the participants’ post-treatment personal and teaching 
efficacy scores are posed in this study. Once the data are collected, the researchers will 
statistically test the hypotheses through the use of multiple regression models. The independent 
variables included in the various regression models represent; (a) the teachers pre-treatment 
personal and teaching efficacy scores, (b) a series of dummy variables that identify the three 
treatment groups, and (c) the product of each of the pre-treatment score variables and each of the 
treatment dummy variables, which are required to test the interaction effects. See Table 1 for a 
list of variables used in the various regression models. 



Insert Table 1 about here 



In this study, the researchers are interested in determining whether the data support the 
following research hj'potheses: 

IH,; An interaction effect between the pre-treatment personal efficacy scores and the 
treatments account for some of the variation in the post-treatment personal efficacy 
scores. 
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2H,: An interaction effect between the pre-treatment teaching efficacy scores and the 
treatments account for some of the variation in the post-treatment teaching efficacy 
scores. 

The full and restricted regression models that will be used to determine if IH, is supported by the 
data are as follows: 

Y, = a,U + b,X, + b^X, + b(,X^, + b 7 X, + b, X, + e , [Model 1] 

Y, = a,U+ b,X, + b.X. + b 4 X 4 + e , [Model 2] 

The probability produced by the F test of the difference between the R' values of Model 1 and 
Model 2 will be used to determine if 1 H, is supported by the data. That is, this probability value 
will be compared to the alpha level set by the researchers. 

The full and restricted regression models that will be used to determining whether 2H, is 

supported by the data are as follows: 

Y, = ajU + b,X, + b,X, + byX, + b.uX,,, + b,, X,, + e, [Model 3] 

Y, = a 4 U + b,X, + b,X, + + e 4 [Model 4] 

The probability of the F test of the difference between the R" values of Model 3 and Model 4 will 
be used to determine if 2H, is supported by the data. Again, this probability value will be 
compared to the alpha level set by the researchers. 

If the researchers find that the data do not support IH,, they will determine whether the 

data support the following additional hypotheses: 

3H|: At least one difference exists among the post-treatment personal efficacy scores of 
the three treatments adjusting for the pre-treatment personal efficacy scores. 
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4Hj: At least one dift'erence exists among the post-treatment teaching efficacy scores of 
the three treatments adjusting for the pre-treatment teaching efficacy scores. 

The full and restricted regression models that will be used to determine whether 3H, is supported 

by the data are as follows; 

Y, = a^U + b,X, + bjX^ + b4X^ + e 2 [Model 2] 

Y, = a5U+ b4X4 + e 5 [Model 5] 

The probability con esponding to the F test of the difference between the values of Model 2 
and Model 5 will be used to determine whether the data support 3H,. This probability value will 

be compared to the alpha level set by the researchers. 

If 3H, is supported by the data, the researchers will determine which adjusted treatment 
means differ. That is, the three pairwise comparisons of the adjusted post-treatment personal 
efficacy means will be conducted. The following regression models will be used to determine 
whether differences exist between the adjusted post-treatment personal efficacy means of 
Treatment 1 versus Treatment 2, Treatment 1 versus Treatment 3, and Treatment 2 v^ersus 
Treatment 3: 

Y, = ajU + b,X, + b^X, + b4X4 + e, [Model 2] 

Y, = a,U+ bjX, + bjXj + b4X4 + e, [Model 6] 

The probability of the t tests of the b, and b, coefficients in Model 2 will be compared to the 
established alpha levels to determine if differences exist between the adjusted post-treatment 
personal efficacy means of Treatment 1 versus Treatment 3 and Treatment 2 versus Treatment j. 
The probability of the t test of the b, coefficient in Model 6 will be compared to the established 




The Responsibility 9 



alpha level to determine if a difference exists between the adjusted post -treatment personal 
efficacy means of Treatment 2 versus Treatment 1. 

In a similar fashion, if 4H, is supported by the data, the researchers will determine which 
adjusted post-treatment teaching efficacy means of the treatments differ. The following 
regression models will be used to determine whether differences exist between the adjusted post- 
treatment teaching efficacy means of Treatment 1 versus Treatment 2, Treatment 1 versus 
Treatment 3, and Treatment 2 versus Treatment 3: 

Y, - a^U + b,X, + b^X, + bjXj + e 4 [Model 4] 

Y 2 = a 7 U+ b,X, + bjXj + bjXj + e^ [Model 7] 

The probability of the t tests of the b, and hj coefficients in Model 4 will be compared to the 
established alpha levels to determine if differences exist between the adjusted post-treatment 
teaching efficacy means of Treatment 1 versus Treatment 3 and Treatment 2 versus 
Treatment 3. The probability of the t test of the b. coefficient in Model 7 will be compared to the 
established alpha level to determine if a- difference exists between the adjusted post-treatment 
teaching efficacy means of Treatment 2 versus Treatment 1 . 

Before statistical testing is conducted in this study, it is important for the researchers to 
identify the number and nature of the statistical tests that will be tested. The 10 statistical tests 
that the hypothetical study contains include the following: 

1. A maximum of five statistical tests could be conducted on the personal efficacy scores. 
A statistical test will be used to test for the existence of an interaction effect between the pre- 
treatment personal efficacy scores and the treatments. If this interaction effect is not significant, 
another statistical test will be used to test for at least one difference among the adjusted post- 
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treatment personal efficacy means of the three treatments. If the test of the adjusted means 
suggests that at least two adjusted means differ among the three treatments, three statistical tests 
of the pairwise comparisons will conducted. 

2. A maximum of five corresponding statistical tests could be conducted on the post- 
treatment teaching efficacy scores. 

The issue facing the researchers is: What alpha levels should be used for the various 
statistical tests'’ The following section of this paper illustrates how a three-step adjustment 
procedure can be used to determine the alpha levels for the individual statistical tests. 

A Three-Step Type I Error Adjustment Procedure 

We take the position that controlling the Type ! error rates in studies that contain multiple 
statistical tests requires the researchers to address three major issues. First, the researcher must 
identify the appropriate conceptual rate error units for which the adjustments will be made. 
Second, once this identification process has been completed, the researchers must determine the 
number and nature of statistical tests included in each error rate unit. Third, the researchers must 
implement a technique that will adjust the alpha level of each statistical test, if necessary, 
contained in the error rate unit. Although there are numerous adjustment methods that could be 
used, we strongly recommend that researchers implement a Bonferroni-type adjustment 
procedure. In spite of the fact that a Bonferroni-type adjustment procedure tends to be more 
conservative, that is, it has less power, than some other types of procedures in certain situations, 
we believe that such an adjustment procedure gives researchers greater flexibility when dealing 
with multiple statistical analyses and complex research designs. 
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Step One: Defining the Error Rate Units 

In order to adjust the Type I error rates for a study that involves multiple statistical tests, 
the researchers must first specify the appropriate conceptual unit for the error rate (Kirk, 1982; 
Hays, 1988; Winer, Brown, & Michels, 1991; Toothaker, 1991). As noted by a number of 
authors, the relative merits of using one conceptual error unit over another can be debated 
(Duncan, 1955; McHugh & Ellis, 1955; Ryan, 1959, 1962; Wilson, 1962), Nevertheless, the 
identification of a conceptual error rate unit, which we find is seldom discussed in research 
studies that include multiple statistical tests, is an important element in the evaluation of a study’s 
results. The selection of the error rate unit forms the logical framework on which the adjustments 
of the alpha levels of the individual statistical tests are based. Researchers should be able to 
clearly identify and justify the formation of the different error rate units. 

In some studies, researchers may be faced with various choices regarding how to define 
the error rate unit. To illustrate this point, again consider the teacher efficacy study. The 
researchers could consider all 10 of the statistical tests as one error rate unit. On the other hand, 
they may identify two error rate units with each unit based on one of the two dependent variables. 
Thus, two possible delineations of error rate units for the hypothetical teaching efficacy study are 
as follows: 

1 . Delineation 1 - The researchers form only one error rate unit for the entire study. 

2. Delineation 2 - The researchers identify two error rate units, which are based on the 
two dependent variables. 

Some error rate units are referred to as experimentwise error units and others are referred 
to as familywise error units. Rather than being overly concerned with whether the unit is 
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experimentwise or familywise, we believe that it is more important that researchers and program 
evaluators clearly identity the error rate units that are being delineated and the number and nature 
of the statistical tests included in each of those units. 

It should be noted that the researchers’ choice of the error rate unit on which the 
adjustments are based could be a factor in determining whether the statistical test results reported 
in a study are statistically significant or not. Thus, analogous to the often suggested practice of 
reporting a R“ value along with the corresponding test of significance as a means of facilitating the 
interpretation of statistical results, we strongly recommend that not only should the error rate 
units be clearly identified in the study, but the researchers should also present the logic on which 
these error rate units are based. 

Step 2: Identify the Number and Nature of the Statistic a l Tests in Each Error Rate Un jL 

As revealed by Equations 1 .1 and 1 .2, the probability of committing at least one Type I 
error for a given number of statistical tests contained in a specified error rate unit is determined by 
two factors: (a) the alpha level established for each statistical test (ajncj) and (b) the number of 
statistical tests (m). Before researchers can determine the appropriate alpha level for each 
statistical test, they must determine the number and nature of the statistical test contained in the 
error rate unit. 

In the teacher efficacy study, the researchers would identify the number and nature of the 
statistical tests contained in the two different groupings of error rate units as follows. 

1 Delineation 1 -- Since only one error rate unit is specified under this identification 
process, all 10 statistical tests included in this study are contained in this error rate unit. These 
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1 0 tests include two statistical tests of interaction effects, two statistical tests of the treatment 
main effects, and six statistical tests of the pairwise comparisons of the adjusted treatment means. 

2 . Delineation 2 - Since this delineation contains tvyo error rate units that are based on 
the two dependent variables, each of these two error rate units contains five statistical tests. The 
error rate unit based on the personal efficacy scores contains the statistical test of the interaction 
effect, the statistical test of the treatment main effect, and the three statistical tests of the pairwise 
comparisons of the adjusted treatment means. The second error rate unit, which is based on the 
teaching efficacy scores, contains five statistical tests that correspond to the tests contained in the 
first error rate unit. 

Once the number and the nature of statistical tests contained in each error unit has been 
identified, the alpha level for each statistical test contained in a given error unit should be adjusted 
in order to prevent the overall alpha level of that error rate unit from becoming inflated to an 
unacceptable level. This task is addressed in the third and final step of this three-step adjustment 
procedure. 

Step 3: Adjusting the Alpha Levels of the Various Statistical Tests. 

We suggest researchers and program evaluators will find that an adjustment approach 
based on the Bonferroni inequality will provide them with a very practical and robust method of 
adjusting the alpha level for each statistical test in a given error rate unit. To illustrate how the 
adjustment procedure can be implemented, assume that the researchers want to maintain the alpha 
level near the .05 for each of the error rate units identified by the researchers in the teacher 
efficacy study. The alpha levels for the individual statistical tests would be calculated as follows 
for the two delineations of error rate units previously discussed: 
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1 . Delineation 1 -- The researchers identified only one error rate unit in this approach and 
that unit contained two statistical tests of the interaction effects, two statistical tests of the 
treatment main effect, and six statistical tests for the pairwise comparisons of the adjusted 
treatment means. Thus, the alpha levels for the statistical tests of the two interaction effects and 
the two treatment main effects tests will be set at .0125, which is obtained by dividing .05 by four. 
If either of the statistical tests of the treatment main effects is significant, the alpha levels for each 
pairw'ise comparison test is set at .0021, which is calculated by dividing the adjusted alpha level 
used for both treatment main eifect statistical tests (.0125) by the number of pairwise comparisons 
of the adjusted treatment means (6). 

2. Delineation 2 — The two error rate units identified in this approach are based on the 
two dependent variables, i .e., the two types of efficacy scores. Each error unit contains one 
statistical test of the interaction effect, one statistical test of the treatment main eifect, and the 
statistical tests of the three pairwise comparisons of the adjusted treatment means. Thus, the 
alpha level used to statistically test the interaction eifect and the treatment main effect for each 
dependent variable is .025, which is obtained by dividing the .05 alpha level by the total number of 
interaction effects (1) and treatment main effects (1) contained in the error rate unit. The alpha 
level for each pairwise comparison test of adjusted treatment means is set at .0083, which is 
obtained by dividing the adjusted alpha level for the treatment main effect (.025) by the number of 
pairwise comparisons of the adjusted treatment means contained in the error rate unit (3). 

Once the alpha levels are adjusted, the researchers would simply compare the probability 
of a given statistical test to its adjusted alpha level to determine if the null hypothesis should be 
rejected. Implementing this three-step adjustment procedure not only provides protection against 
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inflated Type I error rates in studies that involve multiple statistical tests, but it will also forces 
researchers to reflect on the rationale that they use to make those adjustments. 

Two Additional Issues 

It should be noted that two additional issues, which we did not specifically address in the 
examples discussed thus far in this paper, should be part of the reflection process that one uses to 
adjust Type I error rates. The first issue deals with the question: Are your statistical tests 
planned, i.e., do your statistical tests come from a strong theoretical, empirical and/or logical 
base? The second issue deals with the question: Are your statistical tests orthogonal? 
Consideration of these two questions by the researchers is also an essential element in the 
adjustment process of Type I error rates. 

To illustrate this point with respect to planned tests, assume that we decided to form one 
error unit in the hypothetical teacher efficacy study, i.e.. Delineation 1 is used. In addition, we 
have a theoretical reason for assuming that interaction effects will exist for both the personal and 
teaching efficacy scores. In this case, we may decide not to adjust the alpha levels for the two 
statistical tests of the interaction effects, but we would adjust the alpha levels for the subsequent 
statistical tests, which are the statistical tests of the treatment main effects and the pairwise 
comparisons of the adjusted treatment means. Thus, the alpha level for each interaction effect 
would be set at .05. The alpha levels for the two treatment main effects, however, would be set at 
.025, which is obtained by dividing .05 by the number of treatment effects (2). The alpha levels 
for each pairv.'ise comparison of the adjusted treatment means would be set at .004, which is equal 
to the alpha level used for the group main effect (.025) divided by the number of pairv.'ise 
comparisons being tested (6). 
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To further illustrate this point, assume we identify two error rate units such as those 
presented in Delineation 2. Also assume that empirical evidence exists that would allow us to 
expect that interaction effects will be present for both the teaching and personal efficacy scores. 
Since we expected statistically significant interaction effects, we would set the alpha level for each 
test of the interaction effects at .05. Due to the fact that each error rate unit contains only one 
treatment main effect and the interaction effect test contained in that error rate unit is not part of 
the adjustment process, the alpha level for each treatment main effect would be set at .05. In both 
error rate units the alpha level for each pairwise comparison test of the adjusted treatment means 
would be set at .017, which is obtained by dividing the alpha level of the treatment main effect 
(.05) by the number of pairwise comparisons (3) contained in each error rate unit. 

Summary 

To protect against inflated Type I error rates in studies that contain multiple statistical 
tests, researchers need to consider implementing an adjustment process. The three-step 
adjustment procedure presented in this paper, which is based upon a Bonferroni-type adjustment 
process, first requires the researchers to identify the error rate unit or units (pairwise, familywise, 
and experiment wise) that will serve as the basis for the adjustment process. Second, they must 
identify the number and nature of the statistical tests contained in each error rate unit. Finally, 
they must adjust the alpha levels of some, if not all, of the individual statistical tests contained in 
the error rate unit. . 

This three-step adjustment procedure provides researchers with a tool that is robust, 
flexible, and easy to apply. More importantly, however, this procedure requires researchers to 
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reflect on the selection of the conceptual error rate unit on which the Type I error adjustments are 
based. Such reflection should lead to better research and program evaluation. 

In addition to encouraging the use of this three-step adjustment approach, we believe that 
it is important for educational researchers to engage in philosophical discussions and research to 
identify the most appropriate error rate units for specific types of research questions and 
situations. We believe that this type of investigation may prove to be Just as valuable, if not more 
so, for the fields of educational research and program evaluation than additional Monte Carlo and 
analytical studies on Type 1 error rate correction procedures. 
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Table 1 

Description of the Variables and Symbols Used in the Regression Models 



Symbol 


Description 


Y. 


Post-Treatment Personal Efficacy Scores 


Y^ 


Post-Treatment Teaching Efficacy Scores 


X. 


Treatment 1 (If in Treatment 1, X, = 1, otherwise X, = 0) 


X^ 


Treatment 2 (If in Treatment 2, X 2 = 1 ; otherwise X 2 = 0) 


X 3 


Treatment 3 (If in Treatment 3, X 3 = 1; otherwise X, = 0) 


X 4 


Pre-Treatment Personal Efficacy Scores 


X 5 


Pre-Treatment Teaching Efficacy Scores 


Xo 


X, * X, 


X 7 


X3 * X, 


X* 


X3 * X, 


X, 


X, *X, 


X.o 


X3 * X 5 


Xn 


X3 * X5 


U 


unit vector 


^1-7 


constant term 


^1-7 


error terms for the various regression models 
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